<html>
<head><meta charset="utf-8"><title>Async download threads · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Async.20download.20threads.html">Async download threads</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="174162393"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Async%20download%20threads/near/174162393" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Paul Faria <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Async.20download.20threads.html#174162393">(Aug 26 2019 at 17:53)</a>:</h4>
<p>Async question here: I'm trying to download a very large number of files from a remote server. I've already got async/await setup, but spawning a new tokio task clearly won't work because I'll run out of file handles (already confirmed this).</p>
<p>If I wasn't using async/await, I'd just spawn a dedicated number of download threads and use mpsc to send the downloads to the threads with a cycling index and have the threads process their downloads sequentially.</p>
<p>With tokio and async/await, it's not clear how to use their existing thread pool to do something similar.</p>



<a name="174162428"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Async%20download%20threads/near/174162428" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Paul Faria <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Async.20download.20threads.html#174162428">(Aug 26 2019 at 17:53)</a>:</h4>
<p>Do I need to run a runtime in each thread or is there a way to leverage the existing thread pool in a clean way?</p>



<a name="174163553"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Async%20download%20threads/near/174163553" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Taylor Cramer <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Async.20download.20threads.html#174163553">(Aug 26 2019 at 18:07)</a>:</h4>
<p>To clarify, you're trying to limit the number of files that are downloaded at the same time?</p>



<a name="174171096"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Async%20download%20threads/near/174171096" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Paul Faria <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Async.20download.20threads.html#174171096">(Aug 26 2019 at 19:46)</a>:</h4>
<p>Yes (sorry for the really slow response)</p>



<a name="174175138"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Async%20download%20threads/near/174175138" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Taylor Cramer <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Async.20download.20threads.html#174175138">(Aug 26 2019 at 20:40)</a>:</h4>
<p>no problem!</p>



<a name="174175163"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Async%20download%20threads/near/174175163" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Taylor Cramer <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Async.20download.20threads.html#174175163">(Aug 26 2019 at 20:41)</a>:</h4>
<p>You might consider using the <code>.buffered</code> combinator on streams or the limit to <code>for_each_concurrent</code> to limit the number of running futures actively pulling things from the remote server</p>



<a name="174175246"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Async%20download%20threads/near/174175246" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Taylor Cramer <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Async.20download.20threads.html#174175246">(Aug 26 2019 at 20:42)</a>:</h4>
<p>something like <code>stream::iter(vec![thing1, thing2, thing3]).for_each_concurrent(MAX_CONCURRENT_FILES, |thing| async move {... /* get your file info here *' ... })</code></p>



<a name="174175510"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Async%20download%20threads/near/174175510" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Paul Faria <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Async.20download.20threads.html#174175510">(Aug 26 2019 at 20:47)</a>:</h4>
<p>Ok, I was also looking at ThreadPool in futures while waiting for your response, but I guess that's not the right idea. I saw on <a href="http://vorner.guthub.io/async-bench.html" target="_blank" title="http://vorner.guthub.io/async-bench.html">vorner.guthub.io/async-bench.html</a> that futures-cpupool could process multiple requests while limiting IO polling to the main thread (I should have looked at the actual impl). I wasn't sure if ThreadPool was the same idea. I'll code up the <code>for_each_concurrent</code> now. Thanks!</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>